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Abstract 

From the present limits on the masses and mixings of fourth family quarks, they are expected to 
have mass larger than the top quark and allow a large range of mixing of the third family. They 
could also have different dynamics than the quarks of three families of the Standard Model. The 
single production of the fourth family up type quark t' has been studied via anomalous production 
process pp — > t'VX (where V = g, Z, 7) at the LHC with the center of mass energy of 7 and 14 TeV. 
The signatures of such process are discussed within both the SM decay modes and anomalous decay 
modes of t' quarks. The sensitivity to anomalous coupling k/K = 0.004 TeV -1 can be reached at 
y/s = 14 TeV and L int = 100 pb" 1 . 
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I. INTRODUCTION 



The number of fermion families in the Standard Model (SM) and its very extensions 
remain arbitrary. There is a number of experimental discrepancies with the SM expectations 
at some level of standard deviations, and they can also be interpreted as the loop effects or the 
flavor changing neutral current (FCNC) effects by the fourth family fermions. A minimal 
framework can give rise to FCNC effects through exchange of massive SM bosons whose 
couplings to the light fermions get modified by mixing with the fourth family fermions. It 
is also important to interpret the low energy measurements for the contributions from extra 
fermions and analyze them in view of the latest electroweak precision data Qj, 

The current limits on the masses of the fourth SM family leptons are |3j: my > 100 
GeV, m v i > 90 (80) GeV for Dirac (Majorana) neutrinos. Recently, the Collider Detector 
at Fermilab (CDF) has constrained the masses of the fourth SM family quarks: m t i > 335 
GeV at 95% CL. [4], m v > 338 GeV at 95% CL. [5]. On the other hand, the perturbativity 
of the Yukawa coupling implies that m t i < 600 GeV, and the arguments based on the 
partial wave unitarity restrict to an upper bound m t i w 500 GeV [6]. The parameter space 
of fourth family fermion masses with minimal contributions to the oblique parameters S 
and T, and in agreement with all experimental constraints, leads to a quark mass splitting 
m t i — my ~ 50 + 10 \n(mh/115GeV) where mh is the mass of Higgs boson. The heavy 
quark mass bounds may be relaxed if they are considered together with the fourth family 
leptons, which could help in the partial cancellation of the effect of fourth family on the S 
and T parameters. Even in the assumption of the absence of this cancellation the precision 
measurements restrict my — m u > ^ 50 — 100 GeV. A review of theoretical and experimental 
motivations can be found in 

If the fourth family quarks are discovered at the LHC, it is expected that it plays a 
crucial role solving some of the currently open questions such as the CP violation and flavor 
structure of the standard theory j^Jjj], electroweak symmetry breakin g jl6 -19|. hierarchies 
of fermion mass spectrum and mixing angle in quark/lepton sectors |20M23|. The fourth 
family quark masses and mixings can be predicted in the allowed parameter space which is 
in the reach of the LHC. In the four families scenario, the assumption of the unitarity of 4 x 4 
mixing matrix holds and it allows a large range of the elements as far as the magnitudes and 
phases of the V tq are concerned. A serious contribution can be expected for the production 
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of the fourth family fermions from the anomalous interactions. These effects of the fourth 



family quarks have been studied in detail at future linear colliders 
and recently at hadron colliders 



ep colliders 



25 



26|], 



271 ] . The anomalous resonant production of fourth family 



up type quarks via gqi — > t' [qi = u, c) subprocess 28|, |29| and single production of fourth 



family quarks 



have been studied at the LHC 
In this study, we investigate the production of fourth family up type quark (t') with the 
association of the gauge bosons (g, Z, 7) through anomalous interactions at the LHC. Then, 
we consider t' decays into SM and anomalous modes for some relevant parameter space. We 
discuss the decay modes for some parameter space and the t' signal observability at three 
different energies 7 and 14 TeV. 



II. INTERACTIONS WITH FOURTH FAMILY QUARKS 



The interaction Lagrangian for the fourth family quarks (Q' = t', b') within the SM is 
given by 



-g e E QM^Q'^-gs E QiT^Q'fil 

Q'i=t',b' Q\=t'V 



A Q\=VV 

g 



E ^W(l-75)g J ^ ± + H.c. (1) 



sin6 w Qi =t i b i 

where A^, G^, and are the fields for photon, gluon, Z-boson and W ± bosons, 
respectively. T a = X a /2 are Gell-Mann matrices; Q q is the charge of the quark; g e , g z and g s 
are the electromagnetic, weak neutral-current and strong coupling constants, respectively. 
The coupling g z = g e j sin 6\y cos 9y/ where 9w is the Weinberg angle. The g v and g a are the 
vector and axial-vector couplings of the neutral weak current with fourth family quarks. The 
elements of the extended CKM mixing matrix are presented by V^-. The effective Lagrangian 
for the FCNC anomalous interactions between the fourth family quarks, ordinary quarks and 
the gauge bosons V (g, Z, 7) can be written as follows 



E ^-Q^et'wiF^ + E 

qi=u,c,t qi=u,c,t 
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Table I: The branching ratios BR(%) and total decay widths (T) of t' quark depending on the mass 
(nv), here we take the anomalous coupling k/A = 0.1(1) TeV -1 . 



BR(%) ■-> 


Wb 


Ws 


Wd 


gu/gc 


gt 


Zu 1 Zc 


Zt 




7« 


T(GeV) 


300 


72.00(7.40) 


19.00(2.00) 


0.02(0.003) 


3.60(37.00) 


1.00(11.00) 


0.22(2.20) 


0.02(0.24) 


0.08(0.82) 


0.02(0.24) 


0.59(5.75) 


400 


71.00(6.80) 


19.00(1.80) 


0.03(0.003) 


3.50(33.00) 


1.90(18.00) 


0.23(2.20) 


0.10(0.95) 


0.08(0.74) 


0.04(0.39) 


1.43(15.02) 


500 


71.00(6.40) 


19.00(1.70) 


0.03(0.003) 


3.50(32.00) 


2.30(21.00) 


0.23(2.10) 


0.15(1.30) 


0.08(0.70) 


0.05(0.48) 


2.82(30.98) 


600 


71.00(6.20) 


19.00(1.70) 


0.03(0.003) 


3.50(31.00) 


2.60(23.00) 


0.23(2.10) 


0.17(1.50) 


0.08(0.68) 


0.06(0.52) 


4.89(55.27) 


700 


70.00(6.10) 


19.00(1.60) 


0.03(0.003) 


3.40(30.00) 


2.80(25.00) 


0.24(2.10) 


0.19(1.70) 


0.08(0.67) 


0.06(0.55) 


7.78(89.52) 


800 


70.00(6.00) 


19.00(1.60) 


0.03(0.003) 


3.40(30.00) 


3.00(26.00) 


0.24(2.00) 


0.20(1.70) 


0.08(0.66) 


0.07(0.57) 


11.64(135.40) 



+ E 

qi=u,c,t 



K 



fg s t'a^T a q t G^ + h.c. 



+ E ~a~ Qqi9eb' a fj^qiF^ 

qi=d,s,b 



E U-gJwiZ 1 " 

qi=d,s,b 



+ E 

qi=d,s,b 



K 



fg s ba^T aqi G^ + h.c. 



(2) 



where F^, Z^, and G^ are the field strength tensors of the photon, Z boson and gluons, 
respectively; the K 7 , k z and K g define the strength of the anomalous couplings for the neutral 
currents with the photon, Z boson and gluon, respectively. A is the scale for new physics. 



We have implemented both the SM and anomalous interaction vertices for the fourth 



'amily 



quarks into the CompHEP 3l| and we use parton distribution library CTEQ6M 32] with 
the factorization scale Q = mt>- 

Here, we have used the parametrization in which the values for the elements | Vfd \= 0.0044, 
I Vfb |= 0.22 and | Vf s \= 0.114 are given by Ref. [33]. In order to reduce free parameters we 
assume that the anomalous couplings and are equal to k. The analytical expressions 

28|. The 



for the decay widths of t' quark both in SM and anomalous modes are given in Ref. 
branching ratios into both SM and anomalous decay modes are given in Table |U It is seen from 
Table H] that the anomalous decay mode of the t' — > qg is dominant for k/A = 1 TeV -1 . 

The Feynman diagrams for the single production of t' quark together with the vector 
boson V (g, Z, 7) are presented in Fig. [TJ Here, the last two diagrams are relevant only for 
the t'g production process. The analytical expressions of the differential cross sections for 
the subprocess qg — > t'V (V = g, Z, 7) are calculated and given in the Appendix. 



9— — r y/Z/g 

g J— » {/ 

(d) 

S T— » f 



Figure 1: Feynman diagrams for anomalous t'V (1/ = g,Z,j) production at the LHC. 
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Figure 2: The cross sections for anomalous t'V (V = g, Z, 7) production at the LHC with 14 TeV. 

III. NUMERICAL CALCULATIONS 

The total cross sections for the t'V {V = g, Z,j) production with k/A = 0.1 TeV^ 1 are 
presented in Fig. [2] depending on the mass my at the collision center of mass energy of 14 
TeV. As seen from Fig. [2] the cross section for the process pp — > t'gX is much larger than 
that of the pp — > t'jX and pp — » t'ZX processes. Due to large QCD background in the qgg 
production (which includes only one type anomalous coupling n g for the case t' — > W + q) and 
small branching ratio into qj/Z for k/A < 0.1 TeV -1 as seen from Table HJ we consider only 
the processes pp — > t'ZX and pp — > t >r yX with subsequent decay into SM mode t' — > W + b 
for both processes, and the anomalous mode t' — > tg for the first process. Hereafter, in the 
numerical calculations we use the anomalous coupling k/A = 0.1 TeV -1 . 




, pp->t'gx 

k/A=0.1 TeV" 1 pp->t'ZX 

pp-->t'yX 
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Figure 3: The pt distributions of b quark from signal and background process pp — > W + bZX at 
^=14 TeV. 
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Figure 4: The r] distributions of b quark from signal and background process pp — > W + bZX at 
^=14 TeV. 

Transverse momentum distributions of the b quark for the signal pp — > W + bZX (where 
m t i =350, 550, 750 GeV) and the SM background are shown in Fig. |3j The signal differential 
cross sections show peak around pt = 75, 150 and 250 GeV for the values of my = 350, 
550 and 750 GeV, respectively. The background differential cross section smoothly decreases 
with respect to the pr of b quark. The rapidity distributions of the b quark from the signal and 
background locate mostly in the range of \rj b \ < 2 as shown in Fig. [U The signal has peak around 
the t' mass values of 350, 550 and 750 GeV over the smooth background. The level of the 
relevant background is compared with the signal in the invariant mass distributions of W + b 
system as shown in Fig. [5j 
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Figure 5: Invariant mass distributions of the W + b system (pp — > W + bZX) for the given 
parametrization at y/s—14 TeV. 
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Figure 6: Invariant mass distributions of the t + jet system (pp — > tgZX for signal and pp —> tjZX 
for background) at -y/i=14 TeV. 

The differential cross sections for the signal process pp — > tgZX in the tg invariant mass 
distributions are given in Fig. El The signal shows itself as the peaks around the assumed 
mass values of m t >. Here, we take into account the process pp — > tjZX for background 
where j denotes light quarks in the final state. 

The px distributions of the photon from the signal and background processes pp — > 
W + b , yX are shown in Fig. [71 

In order to calculate the statistical significance (SS) we use the following formula 

SS= y/2[(S + B) \n(l + S/B) - S] 
where S and B are the number of events of the signal and background, respectively. In 
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Figure 7: The px distributions of photon from the signal and background at yfs = 14 TeV. 

the analysis, we use the kinematical cuts on the transverse momentum pj, > 50 GeV of 
jets and pj, > 20 GeV of photons. We calculate the signal and background cross sections 
for the process pp — > W + b(Z, 7) X in the invariant mass interval satisfying the conditions 
\m t > - M Wb \ < 10 GeV for the t' mass range 300-500 GeV and \m t , - M Wb \ < 20 GeV 
for the t' mass range 500-800 GeV. In the significance calculations for the signal process 
pp — > W + b(Z, , ~f)X , we take into account the hadronic W- decay and leptonic Z-decay. We 
use 6-tagging efficiency as 0.5 for considered processes. 

The SS values, total cross sections of the signal and background process pp — > W + bZ( , ~f)X 
are given in Table HHY]) for y/s=U TeV and L = 10 5 pb _1 , and in Table [1111© for ^=7 
TeV and L = 10 3 pb" 1 , respectively. 

In Tables I VI I and IVHl we consider leptonic (hadronic) decays of Z-boson and hadronic 
(leptonic) decays of Vy-boson for the process pp — > tjZX. The conditions \m t / — M t j\ < 10 
GeV for m t > =300-500 GeV and \m t > - M tj \ < 20 GeV for m t > =500-800 GeV are used for 
the process pp — > tjZX. Taking the integrated luminosity L = 10 5 pb -1 , the SS values and 
the total cross sections of the signal and background are given in Table [VTl for a/s=14 TeV. 

In Fig. [HI we plot the contours for different mass values, namely =350, 450 and 550 
GeV, with the 3a significance for process pp — > W + bZX. We find that the lower limit on the 
anomalous couplings K g /A and K z /A can be as low as 0.015 and 0.0035 TeV -1 , respectively. 

In Fig. [91 we plot the contours for the mass values m t > = 350, 450 and 550 GeV for 3<r 
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Table II: The total cross sections of the signal and background process pp — > W + bZX, as well as 
the 55 values at yfs=U TeV with L = l^pb -1 . 



ran (GeV) 


in- 5 x an(-ob) 


10 _1 X (TQ(r>h) 


55 




4 85 


fi 7D 


1 65 


400 


2.21 


5.42 


153 


500 


1.05 


4.38 


142 


600 


1.10 


3.82 


131 


700 


0.62 


3.15 


121 


800 


0.37 


2.56 


111 



Table III: The same as Table U but for ^=7 TeV and L = lO^b -1 . 



m t /(GeV) 10- 6 xa B (pb) 10" 1 x <7 5 (pb) 55 



300 


8.54 


1.23 


7.1 


400 


3.56 


0.86 


6.1 


500 


1.56 


0.62 


5.3 


600 


1.50 


0.47 


4.6 


700 


0.77 


0.34 


4.0 


800 


0.41 


0.25 


3.4 



significance. It is found that lower limit on the anomalous couplings n g /A and k 7 /A from 
the process pp — > W + b , yX can be as low as 0.004 and 0.025 TeV -1 , respectively. 

IV. CONCLUSIONS 

The anomalous FCNC interactions could be important for some parameter regions due to 
the expected large masses of the fourth family quarks. From the analysis of single production 
of t' quark associated with the neutral gauge bosons the sensitivity to the (n g , k z ) and k 7 ) 
can be obtained as low as (0.015, 0.0035 ) and (0.004, 0.025) at the LHC with the center of 
mass energy of 14 TeV and the integrated luminosity of 10 5 pb _1 . 
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Table IV: The SS values, total cross sections of the signal and background process pp — > t*yZX at 
Vs=14 TeV with L = K^pb -1 . 



m t , (GeV) 


10~ 6 x a B {pb) 


10 _1 x cr<?(pb) 


SS 


300 


24.6 


2.68 


389 


400 


9.87 


3.12 


446 


500 


4.43 


3.03 


457 


600 


4.36 


2.75 


434 


700 


2.26 


2.51 


426 


800 


1.25 


2.28 


415 



Table V: The same as Table HVl but for TeV and L = 10 3 pb 1 . 



mt'(GeV) W- 7 xa B {ph) 10" 2 x a s {pb) SS 



300 


43.5 


7.45 


21.1 


400 


15.5 


8.30 


23.6 


500 


6.18 


7.42 


23.2 


600 


5.17 


6.13 


21.1 


700 


2.44 


5.05 


19.7 


800 


1.19 


4.03 


17.9 



Table VI: The total cross sections of the signal and background process pp — > tgZX and SS values 
which corresponds to the leptonic (hadronic) decays of Z-boson and hadronic (leptonic) decays of 
W-boson at ^=14 TeV with L = 10 5 pb _1 . 



m^GeV) 


10~ 3 x a B (pb) 


10~ 3 x <r s (pb) 


SS 


300 


0.75 


7.94 


8.0(14.3) 


350 


1.43 


11.0 


8.7(15.7) 


400 


2.19 


12.3 


8.5(15.2) 


500 


3.85 


12.2 


7.1(12.8) 


600 


5.46 


12.2 


6.4(11.5) 


700 


6.94 


10.4 


5.1(9.2) 


800 


8.18 


8.60 


4.1(7.3) 
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Table VII: The same as Table IVT1 but for yfs=7 TeV. 



mf(GeV) 10~ 4 x a B (pb) 10~ 3 x a s (pb) SS 



300 


1.82 


1.52 


3 3f5 9") 


OlIU 


^ 9fl 


9 nn 


o.oyv.o j 


400 


4.59 


2.06 


3.3(5.9) 


500 


7.14 


1.76 


2.5(4.5) 


600 


8.96 


1.55 


2.1(3.7) 


700 


10.1 


1.15 


1.5(2.8) 


800 


10.6 


0.84 


1.1(2.0) 
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Figure 8: The 3a contour plot for the signal observability at y/s = 14 TeV and L = 10 5 pb 



APPENDIX 



The differential cross section for the subprocess qg — > t /r y is given as 



da , . , 

at 



2na s a f K 2 2s[m 2 (m 2 - 6s) + s 2 ] 2n 2 ( 



m 2 — s"* 2 



+ 



+ 



27s 2 A 2 \ [(s - m 2 ) 2 + T 2 m 2 ] 
3(2s 3 - m 6 ) - m 2 [2s(s - 8m 2 ) + i(3m 2 - a)]) 



(s + m 2 ) 



[(s — m 2 ) 2 + r 2 m 2 ] 



+ 2 



K 2 n(m 2 - t) Kg K y [3m 4 + (3u - 7t)m 2 + i(4i - &)] 



t — m 2 



f K 2 i[m 2 (n - 2t) - iu] 
(t — m 2 ) 2 

+ [i 4+ u] 
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Figure 9: The same as Fig. [HJ but for the anomalous couplings k 7 /A vs K g /A. 



+ 



+ 



+ 



s(t — m?){u — m 2 ) ui[(s — m 2 ) 2 +Y 2 m 2 
K 1 K g (s — m 2 ) r3m 4 (n - m 2 ) + 3(s 2 + 3si + i 2 )m 2 + stu 



m 2 (s — m 2 )[m 2 (s — m 2 ) + 2ut\] 



[(§ — m 2 ) 2 + r 2 m 2 ] 



(t — m 2 ) 



3m 4 (t - s) - 3(s 2 + st- t 2 )m 2 + stu 



(3) 



{u — m 2 ) 

where s, t and u are the Mandelstam variables for the subprocess; m and V denote the 
mass and decay width of t' quark, respectively. The strong and electromagnetic coupling 
constants are presented by a s and a, respectively. The differential cross section for the 
subprocess qg — > t'g is given as 



da. i x vra 2 K 2 f 32s[m 2 (m 2 -6s) + s 2 l 32(m 2 - s) 2 . 2 . 

144( m 2 _ |)3 9[16m 6 + 18m 4 (s - m 2 ) + 2m 2 (u 2 + t 2 ) + 3nt(5m 2 + s) 



til 



+ 



[(s — m 2 ) 2 + r 2 m 2 ] 

-8[m 2 (n - 2t) - til] 9[4ts + u(3m 2 + u) - 4m 4 ] + 8(m 2 - £)« 
- (t-m 2 ) 2 + t 
9[8m 6 — 6(2s + u)m 4 + As 2 (m 2 — u) + 9sum 2 — u 2 (2m 2 — §)] 

Asu 

8[3m 4 + (3u - 7t)m 2 + i(4t - u)] + 4m 2 ^{m 2 - s)i + s 2 (m 2 - u)(s - m 2 ) 
i — m 2 s t — m 2 t[(s — m 2 ) 2 + r 2 m 2 ] 

[3(m 2 - u)m 4 + (i - 3m 2 )s 2 + (s - 3m 2 )t 2 - 10ism 2 ](s - to 2 ) 
(i — m 2 )[(s — m 2 ) 2 + r 2 m 2 ] 



+ [t o H] 



(4) 
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+ 



+ 



r 9[(5i - 4s)m 4 - As 2 (i + m 2 ) - 3tsm 2 + P(s - 5m 2 )](s - m 2 ). _ 

Tr^ 2^2 21 " +[s^u] 

t[(s — m z ) z + 1 z m A \ 

9[(5i - s)m 4 + 6s 2 (i-2m 2 ) + i 2 (s - 5m 2 ) + s(5s 2 - 9tm 2 )](s - m 2 ) 

u[(s — m 2 ) 2 + r 2 m 2 ] 



+ sou 



The differential cross section for the subprocess qg — > t'Z is given as 

« 8 a I r-,o/„2 , „2v 6 , ,..,1, ,.,2 



-g{qg ->« J = 4325 2 A 2 S 2^ I m 2 [(j _ m 2)2 + r 2 m 2] 



[l%'+5a)K + ^ 4 (^i-s) 



2m|(s + m 2 ) + s 2 (m,| - m 2 + s) + 2sm 2 m 2 z (-160s$ v + 120% + 9)] 



+ 



36k 2 



[2m 4 - m|(s + m 2 + m|) + 2s(s - 2m 2 )] 



324g , „Kz'%('S m 2 ) r 2 /~ , 2\ -I- n 2\ 4i 

+ — , ^ — —[m z (s + m) — s(s — 2m) — m\ 



+ 36 



[(s — m 2 ) 2 + T 2 m 2 ] 

2 

[2m 4 (i - m|) + m 2 (2m\ + (s + 6i)m| - 2i(s + At)) 



{t — m 2 ) 2 



+ i(2t(s + t) - m 2 z (s + 2t))] + 9vK ? Kz [2m|(m 2 - i) + t(m 2 - u)] 



+ 
+ 



t 2 

2n 2 s(m 2 — s) 



Qm 2 z t[(s — m 2 ) 2 + r 2 m 



21 \%gl + 9 2 a )((i - m 2 )(2m| - mH) - stm 2 ) 



+ 44^(44, " 3)?] + 
+ is — uml)] + 



2k : 



s(i — m 2 ) 



t(t - m 2 ) 

2^ 



m 2 (s — m 2 )(2i — m, 2 ^) + m 2 z {m 2 (t 2 + m 2 ) 
(m| - t)(6m 4 - lltm 2 + 5t 2 ) + s£(3m 2 - £)] (5) 



+ 



[3m 4 (u — m 2 ) — s(s(3m 2 — t) + ?>m 2 z (i — m 2 )) 



(t — m, 2 )[(s — m 2 ) 2 + r 2 m 2 ] 
- £(t(3m 2 - s) + m 2 (Ws - 3m|))]j + [i o fi] 

where g v and g a are the vector and axial- vector couplings of the weak neutral current; m z 
is the mass of the Z boson. 
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